Papers by Richard Diehl Martinez
Tending Towards Stability: Convergence Challenges in Small Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Increasing the number of parameters in language models is a common strategy to enhance performance, but smaller models often underperform compared to their larger counterparts due to their reduced representational capacity. |
| Approach: | They use the Pythia model suite to analyse the training dynamics that underlie this phenomenon. |
| Outcome: | The proposed model suite enables us to examine the training dynamics of small models. |
Attention-based Contextual Language Model Adaptation for Speech Recognition (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing language models do not incorporate utterance level contextual information . however, for some domains like voice assistants, additional context provides a rich input signal . |
| Approach: | They propose a method for training neural speech recognition models on text and contextual data. |
| Outcome: | The proposed model reduces perplexity by 7.0% relative over a standard LM . it also improves perxicity by 2.8% relative to a state-of-the-art model for contextual LM. |
Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language models rely on frequency information because they maximize the likelihood of tokens during training. |
| Approach: | They propose a method for quantifying the frequency bias of a language model by assessing sentence-level perplexity with respect to token-level frequency. |
| Outcome: | The proposed method reduces the frequency bias of a language model by inducing a syntactic prior over token representations during pre-training. |
Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research (2025.emnlp-demos)
Copied to clipboard
Richard Diehl Martinez, David Demitri Africa, Yuval Weiss, Suchir Salhan, Ryan Daniels, Paula Buttery
| Challenge: | Recent advances in large language models (LLMs) have enabled strong performance across diverse tasks, but small enough to train on modest budgets. |
| Approach: | They propose a lightweight, modular framework that enables systematic, hypothesis-driven research for small and medium-scale language model development. |
| Outcome: | The proposed framework enables systematic, hypothesis-driven research for small and medium-scale language model development. |